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Abstract. We investigate the performance of parity check codes using the mapping 
onto Ising spin systems proposed by Sourlas. We study codes where each parity 
check comprises products of K bits selected from the original digital message with 
exactly C checks per message bit. We show, using the replica method, that these 
codes saturate Shannon's coding bound for K ^ oo when the code rate K/C is finite. 
We then examine the finite temperature case to asses the use of simulated annealing 
methods for decoding, study the performance of the finite K case and extend the 
analysis to accommodate different types of noisy channels. The connection between 
statistical physics and belief propagation decoders is discussed and the dynamics of 
the decoding itself is analyzed. Further insight into new approaches for improving the 
code performance is given. 



PACS numbers: 89.90.+n, 89.70.+C, 05.50.+q 
1. Introduction 

Error-correction is required whenever information has to be rehably transmitted through 
a noisy environment. The theoretical grounds for classical error-correcting codes were 
first presented in 1948 by Shannon [||]. He showed that it is possible to transmit 
information trough a noisy channel with a vanishing error probability by encoding up 
to a given critical rate Rc equivalent to the channel capacity. However, Shannon's 
arguments were non-constructive and devising such codes turned out to be a major 
practical problem in the area of information transmission. 
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In 1989 Sourlas 0, |^ proposed that, due to the equivalence between addition over the 
field {0, 1} and multiplication over {±1}, many error-correcting codes can be mapped 
onto many-body spin-glasses with appropriately defined couplings. This observation 
opened the possibility of applying techniques from statistical physics to study coding 
systems, in particular, these ideas were applied to the study of parity check codes. 
These linear block codes can be represented by matrices of columns and M rows that 
transform iV-bit messages io M {> N) parity checks. Each row represents bits involved 
in a particular check and each column represents checks involving the particular bit. 
The number of bits used in each check and the number of checks per bit depends on the 
code construction. We concentrate on the case where exactly C checks are performed 
for each bit and exactly K bits compose each check. 

The code rate R is defined as the information conveyed per channel use R = 
H2{fs)N/M = H2{fs)K/C, where H^ifS) = -(1 - /.) log2(l - fS) - IS log,{fs) is 
the binary entropy of the message with bias fg. 
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Figure 1. The encoding, message corruption in the noisy channel and decoding can 
be represented as a Markovian process. The aim is to obtain a good estimative ^ for 
the original message ^. 

In the mapping proposed by Sourlas a message is represented by a binary vector 
^ G {±1}^ encoded to a higher dimensional vector G {±1}^ defined as ^(ij^j2...j^) = 
^41^2 • • -^j/c) where M sets of K indices are randomly chosen. A corrupted version J 
of the encoded message J° has to be decoded for retrieving the original message. The 
decoding process can be viewed as a statistical Bayesian process ^ (see Fig.^. Decoding 
focuses on producing an estimate $, to the original message that minimizes a given 
expected loss ^))p(j|^))p(^) averaged over the indicated probability distributions. 

The definition of the loss depends on the particular task; the simple Hamming distance 
'^{^/O = can be used for decoding binary messages. An optimal estimator 

for this particular loss function is = sign{Sj)p(^s\j) [H? where S is a. N dimensional 
binary vector representing outcomes of the decoding process. Using Bayes' theorem, 
the posterior probability can be written as In p{S | J) = In p{J \ S) + \n p{S) + const. 
Sourlas has shown [Q that for parity check codes this posterior can be written as a 
many-body Hamiltonian: 

In p{S\J)= -f3 n{S) 
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= I3Y,A^J, \{S, + l3nprior{S), (1) 

where /i = (zi, . . . ik) is a set of indices and ^ is a tensor with the properties Afj, G {0, 1} 
and = C Vi, which determines the M components of the codeword j". The 

second term Ti.pnor{S) stands for the prior knowledge on the actual messages; it can 
be chosen as ?iprior('S') = FY^jLiSj to represent the expected bias in the message 
bits. For the simple case of a memoryless binary symmetric channel (BSC), J is 
a corrupted version of the transmitted message where each bit is independently 
flipped with probability p during transmission. The hyper-parameter (3, that reaches an 
optimal value at Nishimori's temperature 0, ^, is related to the channel corruption 
rate. The decoding procedure translates to finding the thermodynamical spin averages 
for the system defined by the Hamiltonian (|l]) at a certain temperature (Nishimori's 
temperature for optimal decoding); as the original message is binary, the retrieved 
message bits are given by the signs of the corresponding averages. 

In the statistical physics framework the performance of the error- correcting process 
can be measured by the overlap between actual message and estimate for a given scenario 
characterized by a code rate, corruption process and information content of the message. 
To asses the typical properties we average this overlap over all possible codes A and 
noise realizations (possible corrupted vectors J) given the message ^ and then over all 
possible messages: 

m = ^(^:J^ (sign(5.))^,,|^^ (2) 

Here sign(S'j) is the sign of the spins thermal average corresponding to the Bayesian 
optimal decoding. The average error per bit is then given by Pe = (1 ~ m)/2. Although 
this performance measure is not the usual physical magnetization (it can be better 
described as a measure of misalignment of the decoded message), for brevity, we will 
refer to it as magnetization. 

From the statistical physics point of view, the number of checks per bit is analogous 
to the spin system connectivity and the number of bits in each check is analogous to the 
number of spins per interaction. Sourlas' code has been studied in the case of extensive 
connectivity , where the number of bonds ^ ~ ^ ^ | ^ scales with the system size. 

In this case it can be mapped onto known problems in statistical physics such as the 
SK [0 {K = 2) and Random Energy (REM) ^ (K—^oo) models. It has been shown 
that the REM saturates Shannon's bound . However, it has a rather limited practical 
relevance as the choice of extensive connectivity corresponds to a vanishingly small code 
rate. 

Here we present an analysis of Sourlas' code for the case of finite connectivity where 
the code rate is non- vanishing, detailing and extending our previous brief reports R [10 . 
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We show that Shannon's bound can also be attained at finite code rates. We study the 
decoding dynamics and discuss the connections between statistical physics and belief 
propagation methods. 

This paper is organized as follows: in Section II wc introduce a naive mean-field 
model that contains all the necessary ingredients to understand the system qualitatively. 
Section III describes the statistical physics treatment of Sourlas' code showing that 
Shannon's bound can be attained for finite code rates ii K ^ oo. The finite K case and 
the Gaussian noise are also discussed in Section III. The decoding dynamics is analyzed 
in Section IV. Concluding remarks are given in Section V. Appendices with detailed 
calculations are also provided. 



2. Naive Mean Field Theory 

2.1. Equilibrium 




Figure 2. Code performance measured by the magnetization m as a function of 

the noise level p as given by the naive mean- field theory at code rate R = 1/2 and 
K — 2,3,4 respectively from the bottom. The long-dashed line indicates PARA- 
FERRO coexistence. Insets: Maximum initial deviation A for convergence at a noise 
level p = 0.1. Top inset: K = 3 and increasing C. Bottom inset: Code rate R = 1/2 
and increasing K. 
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To gain some insight into the code behavior one can start by considering that the 
original message is C,j = 1 for all j (so m = 1 will correspond to perfect decoding) and 
use Weiss' mean-field theory as a first (naive) approximation. The idea is to consider 
an effective field given by (for unbiased messages with F = 0): 

hf = J.Us^ (3) 

{fJ.:jeiJ.} i£fJ.\j 

acting in every site. The first strong approximation here consists in disregarding the 
reaction fields that describe the infiuence of site j back over the system. The local 
magnetization can then be calculated: 

= (tanh {phf))^^^ ^ tanh/3 (hf)^^^ , (4) 

where we introduced a further approximation taking averages inside the function that 
can be seen as a high temperature approximation. Disregarding correlations among 
spins and computing the proper averages one can write: 

m = tanh ((3 C{1 - 2p) m^-^) , (5) 



where p is the noise level in the channel. An alternative way to derive the above equation 
is by considering the free-energy: 

/(m) = -(l-2p)^m^-^. (6) 

The entropic term s(m) is: 

, , 1 + m fl+m\ 1-m fl-m\ 
s(rn) = In (— ) — 1„ (— ) . (7) 

Minimizing this free-energy one can obtain Eq.(^ whose solutions give the possible 
phases after the decoding process. In Fig. ^ we show the maximum magnetization 
solutions m for Eq.(^ as a function of the fiip rate p at code rate R = 1/2 and 
K = 2,3,4. For K = 2 the performance degrades faster with the noise level than in 
the K > 2 case. The dashed line indicates coexistence between paramagnetic (PARA) 
m = and ferromagnetic (FERRO) m > phases. 



2.2. Decoding Dynamics 

In a naive mean-field framework the decoding process can be seen as an iterative solution 
for d^) starting from a magnetization value that depends on the prior knowledge about 
the original message. The fixed points of this dynamics correspond to the minima of 
the free-energy; a specific minimum is reached depending on the initial condition. In 
the insets of Fig.|^ we show, as a measure for the basin of attraction, the maximal 
deviation between the initial condition and the original message A = 1 — mo that allows 
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Figure 3. Graph representing a code. 

convergence to a FERRO solution. At the bottom inset we show the deviation A at 
code rate R = 1/2, increasing values of K and noise level p = 0.1 . An increasing initial 
magnetization is needed when K increases, decoding without prior knowledge is only 
possible for K = 2. The top inset shows X for K = 3, p = 0.1; as C increases (code rate 
decreases), the basin of attraction increases. 

One can understand intuitively how the basin of attraction depends on the 
connectivities by representing the code in a graph with bit and check nodes and looking 
at the mean- field behavior of a single bit node (see Fig.|]). The corrupted checks 
contribute wrong (—1 for the "all ones" message case) values to the bit nodes (m < 1 
in the mean field). Since check node values correspond to a product of i^' — 1 bit values, 
the probability of updating these nodes to the wrong values increases with K, degrading 
the overall performance. On the other hand, if C increases for a fixed K the bit nodes 
gather more information and are less sensitive to the presence of (a limited amount of 
) wrong bits . 

Although this naive picture indicates some of the qualitative features of real codes, 
one certainly cannot rely in its numerical predictions. In the following sections we will 
study Sourlas' codes using more sophisticated techniques that will substantially refine 
the analysis. 

3. Equilibrium 

3.1. Replica Theory 

In the following subsections we will develop the replica symmetric theory for Sourlas' 
codes and show that, in addition to providing a good description of the equilibrium, it 
describes the typical decoding dynamics using belief propagation methods. 

The previous naive "all ones" messages assumption can be formally translated to the 
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gauge transformation |Tl| Si\-^SiC,i and J^^J^ Dje^t 6 that maps any general message to 
the FERRO configuration defined as ^* = 1 Vi. One can then rewrite the Hamiltonian 
in the form: 

n{s) = -Y,A,j^ \{s,-FY.ikSk . (8) 

M k 

With this transformation, the bits of the uncorrupted encoded message are J° = 1 
Vi and, for a BSC, the corrupted bits are random variables with probability: 

V (J,) = (l-p) SiJ,-l)+pS (J^+1) , (9) 

where p is the channel flip rate. For deriving typical properties of these codes one has 
obtain an expression for the free-energy by invoking the replica approach where the 
free-energy is defined as: 



f = lim — — 

^ f3 N^oc N dn 



(10) 



n=0 



where represents an analytical continuation in the interval n G [0, 1] of the 

replicated partition function defined as: 



Tr 



11) 



(12) 



/ /: \ \ Z i - - p 

The magnetization can be rewritten in the gauged variables as : 

m = ((sign(5i))_4__;|g.)^, 

where ^* denotes the transformation of a message ^ into the FERRO configuration. The 
usual magnetization per site can be easily obtained by calculating 

From this derivative one can find the distribution of the effective local fields hj that can 
be used to asses the magnetization m, since sign {{Sj)) = sign.{hj) . 

To compute the replicated partition function we closely follow Ref. |]T2[. We average 
uniformly over all codes A such that = C Vi to find: 



AM 



(13) 



= exp ^ iV Extr, 



q,q 



<=0 {ai...ai) 



C I (lai...aiQai...ai 
\l=0 {ai...a,) 

V'=0("i-"i> 



8 



where % = (tanh {f3J))j, as in [|T3], and go = 1- We give details of this calculation in 
the Appendix A. At the extremum the order parameters acquire expressions similar to 
those of Ref. pl: 



(lai,...,ai 



1 = 



(15) 



vt=l 



where 



c 



(16) 

^1=0 {ai...ai) 

and = Tr{5c.} [{...)X] /Tt{s^} [(...)]• The term ^(5) = E?=oE(ar...ai) qai...aiS''^ . . . 
represents a probability distribution over the space of replicas and Po{S_) = (^e'^^^^a -^"^^ 
is a prior distribution over the same space. For reasons that will become clear in Section 
S Qai,...,ai represents one l-th momentum of the equilibrium distribution of a bit-check 
edge in a belief network during the decoding process and qai...ai represents l-th moments 
of a check-bit edge equilibrium distribution . The distribution X represents the proba- 
bility of a certain site (bit node) configuration subjected to exactly C interactions and 
with prior probability given by pq. 



3.2. Replica Symmetric Solution 

The replica symmetric (RS) ansatz can be introduced via the auxiliary fields 7r(x) and 
vf (y) in the following way (see also fTT 



Qai...ai 



dyn{y) tanh' (/??/), 
dx Tc{x) tanh.\(3x) 



(17) 



for / = 1,2,.... 

Plugging it into the replicated partition function (0), performing the limit n ^ 
and using Eq. (p!Q|) (see Appendix [Appendix B| for details) one obtains: 



/ = — — Extr^^ {a In cosh jd 



-\- a 



K 



W dxi Tc{xi) 



U=l 



In 



K 



l+tanh/9J J^tanh/?, 



C j dx dy 7r(x) 7f(y) In [1 + tanh/?x iaxih. f3y] 
C j dy 7r{y) In cosh (3y 



+ 



c 



n (^yi ^(yi) 
.1=1 



In 



c 



2coshp \ Y,yj + F^ 



where a = C/K. The saddle-point equations, obtained by varying Eq. ([T8| ) with respect 
to the probabihty distributions, provide a set of relations between tt{x) and TT{y) 



7t{x) 



TT 



(y) 



■c-i 

n dyi Tf{yi) 
.1=1 

K-l 

Yl dXi 7!-{xi] 
. 1=1 
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X 



C-l 

1 



(19) 



K-1 



y — — tanh ^ ^tanh/3J tanh/5xj 



Later we will show that this self-consistent pair of equations can be seen as a mean- 
field version for the belief propagation decoding. Using Eq.(|l^) one finds that the local 
field distribution is : 



P{h) 



c 



Y[dyi Tf{yi) 
.1=1 
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c 



(20) 



where nijj) is given by the saddle point equations above. 
The magnetization (|]) can then be calculated using: 



m 



dh sign(/i) P{h). 



(21) 



The code performance can be assessed by assuming a particular prior distribution 
for the message bits, solving the saddle-point equations (|^) numerically and then 
computing the magnetization. 

Instabilities in the solution within the space of symmetric replicas can be probed 
looking at second derivatives of the functional whose extremum defines the free-energy 
(|I^). The simplest necessary condition for stability is having non-negative second 
functional derivatives in relation to 7r(a;) (and 7f(y)) : 



1 



■K-2 

W dxi n{xi) 
. 1=1 



In 



K-2 



1 + tanh.pj tanh^/3x tanh/5, 



>0, 



(22) 



for all X. The replica symmetric solution is expected to be unstable for sufficiently low 
temperatures (large (3). For high temperatures we can expand the above expression 
around small (3 to find the stability condition: 

{J)j{x)^-' > (23) 
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We expect the average (x),r = / dxTT{x)x to be zero in PARA phase and positive 
in FERRO phase, satisfying the stabihty condition. This result is still generally 
inconclusive, but provides some evidence that can be examined numerically. In Section 



3^ we will test the stability of our solutions using condition (|2^). 

In the next sections we restrict our study to the unbiased case (F = 0), which is 
of practical relevance, since it is always possible to compress a biased message to an 
unbiased one. 

3.3. Case K ^ oo, C = aK 

For this case one can obtain solutions to the saddle-point equations for arbitrary 
temperatures. In the first saddle-point equation (0) one can write: 

x=j:m^iC- l){y)^ ={C-1) fdyy n{y). (24) 

It means that if (y)^ = (as it is the in PARA and spin glass (SG) phases) then 7i{x) 
must be concentrated at x = implying that 7r(x) = 6{x) and n{y) = 6{y) are the only 
possible solutions. Moreover, Eg . (plj) implies that in FERRO phase one can expect 
X ^ 0{K). 

Using Eq.(l^) and the second saddle-point equation (p!9D one can find a self- 
consistent equation for the mean-field {y)i;,: 

{yh = (;^tanh-i [tanh(/5J) (tanh(/?(C - l){yh)f-']^ • (25) 

For a BSC the above average is over distribution (^. Computing the average, using 
C = aK and rescaling the temperature as /3 = (3{lnK)/K, in the limit K — oo one 
obtains: 

(y)- = (1 - 2p) [tanh0a{y)- HK))f , (26) 

where p is the channel flip probability. The mean-field (y)^ = is always a solution to 
this equation (either PARA or SG); at (3c = ln{K) /{2aK{l — 2p)) an extra non-trivial 
FERRO solution emerges with (y)^ = 1 — 2p. As the connection with the magnetization 
m is given by Eq. ( pOf ) and Eq. (|2T|) ; it is not difficult to see that it implies m = 1 
for FERRO solution. One remarkable point is that the temperature were the FERRO 
solution emerges is jSc ~ 0{ln{K)/ K); it means that in a simulated annealing process 
PARA-FERRO barriers emerge quite early for large K values implying metastability 
and, consequently, a very slow convergence. It seems to advocate the use of small K 
values in practical applications. This case is analyzed in Section |3.5| . For jS > jSc both 
PARA and FERRO solutions exist. 

The FERRO free-energy can be obtained from Eq.(|T^ using Eq. (P^ , being 
/ferro = — tt(l — 2p). The corresponding entropy is sferro = indicating a single 
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Figure 4. Phase diagram in the plane temperature T versus noise level p for K ^ oo 
and C = Qf-ftT, with a = 4. The dotted line indicates Nishimori's temperature Tjv . 
Full lines represent coexistence. The critical noise level is Pc- The necessary condition 
for stability in the FERRO phase is satisfied above the dashed line. 



solution. The PARA free-energy is obtained by plugging 7r(x) = 5{x) and 7f(?/) = Siy) 
into Eq. (|18D: 

/para = - ln(cosh 13) + In 2), (27) 

spARA = a(ln(cosli 13)- 13 tanh [3) + In 2. (28) 

PARA solutions are unphysical for a > (In 2)/(/? tanh /3 — In ch /3), since the 
corresponding entropy is negative. To complete the phase diagram picture we have 
to assess the spin-glass free-energy and entropy. We have seen in the beginning of 
this section that replica symmetric SG and PARA solutions consist of the same field 
distributions for K ^ oo, implying unphysical behavior. In order to produce a solution 
with non-negative entropy one has to break the replica symmetry. We use here a 
pragmatic way to build this solution, using the simplest one-step replica symmetry 
breaking known as frozen spins. 

It was observed in Ref. that for the REM a one-step symmetry breaking 



scheme gives the exact solution. In this scheme the n replicas' space is divided to 
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groups of m identical solutions. It was shown that an abrupt transition in the order 
parameter from a unique solution (Edwards- Anderson parameter g = 1, SG phase) to 
a completely uncorrelated set of solutions (g = 0, PARA phase) occurs. This transition 
takes place at a critical temperature (3g that can be found by solving the appropriate 
saddle-point equations; this temperature is given by the root of the replica symmetric 
entropy {srs = 0) meaning that the RS-RSB transition occurs at the same point as the 
PARA-SG in this model. The symmetry breaking parameter was found to be = f3g/ j3, 
indicating that this kind of solution is physical only for (3 > (3g, since rrig < 1 [T^, 
indicating a PARA-SG phase transition. The free-energy can be computed by plugging 
the order parameters in the effective Hamiltonian, obtained after averaging over the 
disorder and taking the proper limits. It shows no dependence on the temperature, 
since for (3 > (3g the system is completely frozen in a single configuration. 

For the Sourlas' code, in the regime we are interested in, SG solutions to the saddle- 
point equations are given by 7r(a;) = 5{x) and 7f(?/) = 5{y). The RSB-SG free-energy 
that guaranties continuity in the SG-PARA transition is identical to /para? since the 
SG and PARA solutions have exactly the same structure, to say: 

/rsB-SG = --^ (« In (cosh Pg) + In 2), (29) 

Pg 

where I3g is a solution for srs-sg = (In (cosh [3) — [3 tanh (3) +\q. 2 = 0. 

In Fig.^ we show the phase diagram for a given code rate R in the temperature T 
versus noise level p plane. 

3.4- Shannon's Limit 

Shannon's analysis shows that up to a critical code rate Re-, which equals the channel 
capacity, it is possible to recover information with arbitrarily small error probability for 
a given noise level. For the BSC : 

Rc= — = l+plog2P+(l-p) log2 (1-p). (30) 

Sourlas' code, in the case where K ^ oo and C ~ 0{N^) can be mapped onto 
the REM and has been shown to be capable of saturating Shannon's bound in the 
limit i? 0. In this section we extend the analysis to show that Shannon's bound 
can be attained by Sourlas' code at zero temperature also for ^ oo limit but with 
connectivity C = aK. In this limit the model is analogous to the diluted REM analyzed 
by Saakian in |T^. The errorless phase is manifested in a FERRO phase with perfect 
alignment (m = 1) (condition that is only possible for infinite K) up to a certain critical 
noise level; a further noise level increase produces frustration leading to a SG phase 
where the misalignment is maximal (m = 0). The FERRO- SG transition is analogous 
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to the transition from errorless decoding to decoding with errors described by Shannon. 
A PARA phase is also present when the transmitted information is insufficient to recover 
the original message {R> 1). 



1.0 




-1.4 -1.0 -0.6 -0.2 0.2 0.6 1.0 1.4 

y 



Figure 5. Histogram representing the mean- field distribution Tr{y) obtained by Monte- 
carlo integration at low temperature {(3 = 10, K = 3,C = 6 and p = 0.1). Dotted lines 
represent solutions obtained by iterating self-consistent equations both with five peak 
and three peak ansatze. Inset: detailed view of the weak regular part arising in the 
Monte-carlo integration. 

At zero temperature saddle-point equations (|^) can be rewritten as: 









C-l 


n dyi T({yi) 


6 


X 


- E yj 


.1=1 








-K-l 






Yl dxi it{xi) 
. 1=1 


\[ 


5 


y - sign( 



(31) 



K-l 



y - sign(J Yl xi)mm{\ J |, | xk~i |) 



1=1 



The solutions for these saddle-point equations may, in general, result in probability 
distributions with singular and regular parts. As a first approximation we choose the 
simplest self-consistent family of solutions which are, since J = ±1, given by: 

vf (y) = - 1) + po6{y) + p.6iy + 1) (32) 
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TT X 



C-1 

:=i-c 



0, 



with 



E 

{fc,/i,m} 



L 1\ ^+ ^° ^- 



(33) 



(34) 



where the prime indicates that k, h, m are such that k—h = I; k+h+m = C— 1. Evidence 
for this simple ansatz comes from Monte-carlo integration of Equation (|1^) at very low 
temperatures, that shows solutions comprising three dominant peaks and a relatively 
weak regular part. Inside FERRO and PARA phases a more complex singular solution 
comprising five peaks 7r{y) = {y —I) +p+S{y —0 .5) +po6 (y) +p-6 {y+0.5) +P-2S (y+l) 
collapses back to the simpler three peak solution. In Fig.^ we show a typical result of 
a Monte-carlo integration for the field vf(y). The two peak that emerge by using either 
the three peak ansatz or the five peak ansatz are shown as dotted lines. In the inset we 
show the weak regular part of the Monte-carlo solution. 

Plugging the above ansatz in the saddle-point equations one can write a closed set 
of equations in p± and po that can be solved numerically (see appendix D for details). 

The three peak solution can be of three types: FERRO (p+ > p-), PARA (po = 1) 
or SG (p_ = p+). Computing free-energies and entropies enables one to construct the 
phase diagram. At zero temperature the PARA free-energy is /para = — a and the 
entropy is spara = (1 ~ c^) 2, this phase is physical only for a < 1, what is expected 
since it corresponds exactly to the regime where the transmitted information is not 
sufficient to recover the actual message {R> 1). 

The FERRO free-energy does not depend on the temperature, having the form 
/ferro = — «(1 — 2p) with entropy sferro = 0. One can find the FERRO-SG 
coexistence line that corresponds to the maximum performance of a Sourlas' code by 
equating Eq.(|29|) and /ferro- Observing that Pg = Pn{Pc) (as seen in Fig.^ ) we found 
that this transition coincides with Shannon's bound Eq.(^). It is interesting to note 
that in the large K regime both RS-FERRO and RSB-SG free-energies (for T < Tg) do 
not depend on the temperature, it means that Shannon's bound is valid also for finite 
temperatures up to Tg. In Fig.| we give the complete zero temperature phase diagram. 

The stability of replica symmetric FERRO and PARA solutions used to obtain 
Shannon's bound can be checked using Eg . (p2D at zero temperature: 



lim \ 





-K-2 




1 


W dxi n{xi) 






. 1=1 





K-2 



l + tanh/3J tanh^/3x tanh. (3xj 



>0, 



(35) 



for all X. For PARA solutions the above integral vanishes, trivially satisfying the 
condition, while for FERRO solution in the K large regime, xi ~ 0{K) and the integral 
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Figure 6. Phase diagram in the plane code rate R versus noise level p for K ^ oo 
and C — aK at zero temperature. The FERRO-SG coexistence line corresponds to 
the Shannon's bound. 

becomes 

- 2p [(1 - e (x + 1)) + |x| (0 (x + 1) - e (x - 1)) + e (x - 1)] , (36) 

where G(x) = 1 for x > and otherwise, indicating instabihty for p > 0. For the 
noiseless case p = the stabihty condition is satisfied. The instabihty of FERRO phase 
opens the possibihty that Sourlas' code does not saturate Shannon's bound, since a 
correction to the FERRO solution could change FERRO-SG transition line. However, 
it was shown in Section |3.2| that this instability vanishes for large temperatures, what 
supports, to some extent, the FERRO-SG line obtained and the saturation of Shannon's 
bound in some region, as long as the temperature is lower than Nishimori's temperature. 
For finite temperatures the stability condition for FERRO solution can be rewritten as: 

(l + tanh(/3)tanh2(/?x))^^"''^ (l - tanh(/5)tanh2(/5x))^ > 1 Vx. (37) 

For p = the condition is clearly satisfied. For finite p a critical temperature above 
which the stability condition is fulfilled can be found numerically. In Fig.|^ we show 
this critical temperature in the phase diagram; one can see that there is a considerable 
region in which our result that Sourlas' code can saturate Shannon's bound is supported. 
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Conclusive evidence to that will be given by simulations presented in Section 



3.5. Finite K Case 



Although Shannon's bound only can be attained in the limit K oo, it was shown 
in the Section |3]^ that there are some possible drawbacks, mainly in the decoding of 
messages encoded by large K codes, due to large barriers which are expected to occur 
between PARA and FERRO states. In this section we consider the finite K case, for 
which we can solve the RS saddle-point equations ([T9| ) for arbitrary temperatures using 
Monte-carlo integration. We can also obtain solutions for the zero temperature case 



using the simple iterative method described in Section 3.4 
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Figure 7. Top: zero temperature magnetization m as a function of the noise level 
p for various K values at code rate R — 1/2, as obtained by the iterative method . 
Notice that the RS theory predicts a transition of second order ioi K = 2 and first 
order for K > 2. Bottom: RS-FERRO free-energies (white circles for if = 2 and 
from the left: K = 3,4,5 and 6) and RSB-SG free-energy (dotted line) as functions 
of the noise level p. The arrow indicates the region where the RSB-SG phase starts to 
dominate. Inset: a detailed view of the RS-RSB transition region. 

At the top of Fig.|^ we show the zero temperature magnetization m as a function of 
the noise level p at code rate R = 1/2. These curves were obtained by using the three 
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peak ansatz of the Section |3.4| . It can be seen that the transition is of second order 
for K = 2 and first order for > 3 similarly to extensively connected models. The 
transition as described by the RS solution tends to p = 0.5 as K grows. Note that this 
does not correspond to perfect retrieval since the RSB spin glass phase dominates for 
p > Pc (see bottom of Fig.^. In the bottom figure we plot RS free-energies and RSB 
frozen spins free-energy, from which we determine the critical probability pc where the 
transition occurs (pointed by an arrow). After the transition, free-energies for = 3, 4, 5 
and 6 acquire values that are lower than the SG free-energy; nevertheless, the entropy 
is negative and these free-energies are therefore unphysical. It is remarkable that this 
critical value does not change significantly for finite K in comparison to infinite K. 
Observe that Shannon's bound cannot be attained for finite K, since m = 1 exactly 
only if K oo. 

The K = 2 model with extensive connectivity (SK) is known to be somewhat 
special, a full Parisi solution is needed to recover the concavity of the free-energy and 
the Parisi order function has a continuous behavior [|1^. No stable solution is known for 
the intensively connected model (Viana-Bray model). In order to check the theoretical 
result obtained one relies on simulations of the decoding process at low temperatures. 
In Section VIII we show that the simulations are in good agreement with the theoretical 
results. 

3.6. Gaussian Noise 

Using the replica symmetric free-energy (pIS]) and the frozen spins RSB free-energy (p9|) 
one can easily extend the analysis to other noise types. The general PARA free-energy 
and entropy can be written: 

/para = - ^ (In f3J))j + In 2) 

SPARA = « ((In (ch (3 J)) J - [3{J tanh {[3J))j) + In 2. (38) 
The SG-RSB free-energy is given by : 

/SG-RSB = (« (In (ch (3gJ))j + In 2) , (39) 

with I3g defined as the solution of 

a ((In (ch PgJ))j - (3g{J tanh {(3gJ))j) + In 2 = 0. (40) 
The FERRO free-energy is in general given by /ferro = {J)j = 



—a (J tanh {(3nJ))j (see Appendix [Appendix D]) . The maximum performance of the 
code is defined by the critical line : 

a ((ln(ch PgJ))j - Pg{J tanh(/?^ J)) j) + In 2 = 0, (41) 
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obtained by equating free-energies in PARA and FERRO phases. Comparing this 
expression with entropy ( ^0]) it can be seen that Pg = at the critical hne; the 
same behavior observed in the BSC case. From Eq.(^TD one can write: 



-(log2 cosh(/?J))j 



(42) 



that can be used to compute the performance of the code for arbitrary symmetric noise. 




Figure 8. Critical code rate Rc and channel capacity for a binary Gaussian channel 
as a function of the signal to noise rate S/N (solid line). Sourlas' code saturates 
Shannon's bound. Channel capacity of the unconstrained Gaussian channel (dashed 
line). 

Supposing that the encoded bits can acquire totally unconstrained values Shannon's 
bound for Gaussian noise is given hj = ^ log2(l + S/N), where S/N is the signal to 
noise ratio, defined as the ratio of source energy per bit (squared amplitude) over the 
spectral density of the noise (variance). If one constrains the encoded bits to binary 
values {±1} the capacity of a Gaussian channel is: 

Rc = J dJ P{J I 1) log2P(J j dJ P{J) log2P(J), (43) 
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where P(J | 7°) = ^ exp(-(^). 

In Fig.^ we show the performance of Sourlas' code in a Gaussian channel together 
with the capacities of the unconstrained and binary Gaussian channels. We show that 
K ^ oo, C = aK Sourlas' code saturates Shannon's bound for the binary Gaussian 
channel as well. The significantly lower performance in the unconstrained Gaussian 
channel can be trivially explained by the binary coding scheme while signal and noise 
are allowed to acquire real values. 



4. Decoding Dynamics 

4-1. Belief Propagation 

The decoding process of an error-correcting code relies on computing averages over the 
marginal posterior probability P{Sj \ J) for each one of the N message bits Sj given the 
corrupted encoded bits (checks), where /i = (zi . . . ix) is one of the M sets chosen by 
the tensor A^. The probabilistic dependencies existing in the code can be represented 
as a bipartite graph known as a belief network where nodes in one layer correspond to 
the M checks while nodes in the other to the bits Sj. Each check is connected to 
exactly K bits and each bit is connected exactly to C checks (see Fig.^). 

Pearl |]TB[ proposed an iterative algorithm for computation of marginal probabilities 



in belief networks. These algorithms operate by updating beliefs (conditional 
probabilities) locally and propagating them. Generally the convergence of these 
iterations depends on the absence of loops in the graph. As can be seen in Fig.^, 
networks that define error-correcting codes may include loops and convergence problems 
may occur. Recently it was shown that in some cases Pearl's algorithm works even in 
the presence of loops [|19 . 



The particular use of belief networks as decoding algorithms for error-correcting 



codes based on sparse matrices was discussed by MacKay in pO[. In this work a loop- 
free approximation for the graph in Fig.^ was proposed (see [0 for a general discussion 
on such approximations). In fact, it was shown in |21| that the probability of finite length 



loops in these graphs vanishes with the system size. 

In this framework the network is decomposed in a way to avoid loops and the 
conditional probabilities q'^j and r^^^ are computed. The set of bits in a check ^ is 
defined as C{^) and the set of checks over the bit j as The probability that 

5*^ = 5* given information on all checks other than ji is denoted q^^^ = P{Sj = S \ {J,y : 
V G M{3) \ /i}) and rj,f = Tr|5,;ig£(^)\j}P( | Sj = S,{Si:l e C{fi) \ j}) UiecM\i 
is the probability of the check if the bit j is fixed to Sj = S and the other bits 
involved are supposed to have distributions given by q^^ . In Fig.||b one can see a 

(S) 

graphical representation of rj^^ that can be interpreted as the influence of the bit Sj 
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Figure 9. (a) Belief network representing an error-correcting code. Each bit Sj (white 
circles) is linked to exactly C checks and each check (black circles) is linked to exactly 
K bits, (b) Graphical representation of the field r^y. The grey box represents the mean 
field contribution Y[iec{fj.)\j 9m' "-"^ other bits on the check J^. (c) Representation 
of one of the fields g^;. 



and the mean-field Ylii^c{p)\i Q^d''^ (representing bits in C{jj,) over than /) over the check 
J^. In the Fig.|c we see that each field q^^P represents the infiuence of the checks in 
excluding /i, over each bit Si, this setup excludes the loops that may exist in the 
actual network. 

(S) 

Employing Bayes theorem, can be rewritten as: 

qlf = a,, P({ J, :uEMij)\fi}\ S,) pf\ (44) 

where a^j is a normalization constant such that q^^^^ + q^^j ""^^ = 1 and p^"^^ is the prior 
probability over the bit j. The distribution P{{Ju '■ G A1(j) | Sj) can be replaced 
by a mean-field approximation by factorizing dependencies using fields r[^^- 



^S) _ ^ JS) Yj ^{S) 



u£M{j)\n 

rif = i:^s,ieci,mP{J, I = s,{Sr.ie /:(/x) \ j}) n ^ff"- (45) 

i6£(/.)\i 
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A message estimate = sign y{Sj)^{s)j can be obtained by solving the above 
equations and computing the pseudo-posterior: 

gf = a,pf n (46) 

where Qjj IS cL normahzation constant. 

By taking advantage of the normahzation conditions for the distributions q^^^'' + 
= 1 and r^/^^^ + r^f^j = 1 one can change variables and reduce the number of 
equations (^51) to the couple 5g^j = g^^"*^^ — q^^^^ and 5r^j = r^^^^ — r^ijj^\ Solving these 
equations, one can find back r^^^ = ^ (1 + 5r^jSj) and the pseudo-posterior can be 
calculated to obtain the estimate. 



4-2. Connection with Statistical Physics 

The belief propagation algorithm was shown in to outperform other methods such 
as simulated annealing. In |^ it was proposed that this framework can be reinterpreted 
using statistical physics. The main ideas behind the approximations contained in 
(1^) are somewhat similar to the Bethe p2[ approximation to diluted two-body spin 



glasses. Actually, for systems involving two-body interactions it is known that the Bethe 
approximation is equivalent to solving exactly a model defined on a Cayley tree and that 
this is a good approximation for finitely connected systems in the thermodynamical limit 
23| . In fact, loops in the connections become rare as the system size grows and can be 



neglected without introducing significant errors. The belief propagation can be seen as 
a Bethe-like approximation for multiple bodies interaction systems. 

The mean-field approximations used here are also quite similar to the TAP approach 



23. The fields ql^ correspond to the mean influence of other sites other the site j and 



the fields r^j represent the influence of j back over the system (reaction fields). 

The analogy can be exposed by observing that the likelihood p(J^ | >S') is 
proportional to the Boltzmann weight: 

WBiJ, I {5", : J G C{fi)}) = exp l-f3J, l[sA. (47) 

That can be also written in the more convenient form: 

WBiJ^ I {5", : J e Cifi)}) = ^cosh(/3J^) I 1 + tanh(/?J^) [] ^j] ■ (48) 

(s ■) 

The variable r^j can then be seen as proportional to the effective Boltzmann weight 
obtained by flxing the bit Sji 

WesiJf. I Sj) = ^T{Sr.leCi^.)\j} Wij(J^ \ {Si : I e C{n)}) [] I^V ■ (49) 

ieCM\j 
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Plugging Eq.(^) for the likelihood in equations (^S]), using the fact that the prior 
probability is given by pf^ = | (1 + tanh(/35'F)) and computing 6q^j and 6r^f 



6r^j = tanh(/3J^) ]J 6qf,i 
5q^j = tanh ^ tanh~"'^(5r,^j) + (3F 



(50) 
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Figure 10. Magnetization as a function of the flip probability p for decoding using 
TAP equations for iiT = 2. From the bottom: Monte-carlo solution of the RS saddle- 
point equations for unbiased messages (/s = 0.5) at T = 0.26 (hne) and 10 independent 
runs of TAP decoding for each flip probability (plus signs), T = 0.26 and biased 
messages {fs = 0.1) at Nishimori's temperature T^. 

The pseudo-posterior can then be calculated: 



6qj = tanh ^ tanh ^(5? 



+ (3F 



(51) 



providing Bayes' optimal decoding = sign(5gj). It is important at this point to 
support the mean-field assumptions used here by methods of statistical physics P|. The 
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factorizability of the probability distributions can be explained by weak correlations 
between connections (checks) and by the cluster property: 



a!™o iV2 ^ {(^i^3)p{S\J) - {Si)piS\J){Sj)piS\J) 







(52) 



that bits Sj obey within a pure state |T7 
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Figure 11. Magnetization as a function of the flip probability p for decoding using 
TAP equations for K — h. The dotted line is the replica symmetric saddle-point 
equations Monte-carlo integration for unbiased messages (/^ = 0.5) at the Nishimori's 
temperature Tjv. The bottom error bars correspond to 10 simulations using the TAP 
decoding. The decoding performs badly on average in this scenario. The upper 
curves are for biased messages (/s — 0.1) at the Nishimori's temperature Tjv. The 
simulations agree with results obtained using the replica symmetric ansatz and Monte- 
carlo integration. 



One can push the above connections even further. Eqs.(|50|), of course, depend on 
the particular received message J . In order to make the analysis message independent, 
one can use a gauge transformation (5r^j ^ ^j^^^j and 6q^j to write: 



5r^j = tanh(/3J) Y[ 6q^i 
iec{p)\j 



24 

5q^j = tanh I ^ tanh~^(5r^j) + (3^jF | . (53) 

In this form a success in the decoding process correspond to 5r^j > and 5q^j = 1 
for all n and j. For a large number of iterations, one can expect the ensemble of 
belief networks to converge to an equilibrium distribution where 6r and 6q are random 
variables sampled from distributions p{y) and p{x) respectively. By transforming these 
variables as 6r = tanh(/3?/) and 6q = tanh(/3a;) and considering the actual message and 
noise as quenched disorder, Eqs.(^) can be rewritten as: 

1 / - / 
1/ = — ( tanh ^ tanh(/3J) tanh(/3xi 

P \ V i=i 




/c-i 



,E%+^^ ■ (54) 

The above relations lead to a dynamics on the distributions p{y) and p(x), that is 
exactly the same obtained when solving iteratively RS saddle-point equations (pJ]). The 
probability distributions piy) and p{x) can be ,therefore, identified with nijj) and ti{x) 
respectively and the RS solutions correspond to decoding a generic message using belief 
propagation averaged over an ensemble of different codes, noise and signals. 

Eqs.(|50|) are now used to show the agreement between the simulated decoding 
and analytical calculations. For each run, a fixed code is used to generate 20000 
bit codewords from 10000 bit messages, corrupted versions of the codewords are then 



decoded using (|50D. Numerical solutions for 10 individual runs are presented in Figs.|TO 
and |TI|, initial conditions are chosen as 5r^i = and 6q^i = tanh(/5F) reflecting prior 
beliefs. In Fig.|lO| we show results for K = 2 and C = 4 in the unbiased case, at code rate 
R = 1/2 (prior probability p^p = f = 0.5) at a low temperature T = 0.26 (we avoided 
T = due to numerical difficulties). Solving saddle-point equations (|^) numerically 
using Monte-carlo integration methods we obtain solutions with good agreement to 
simulated decoding. In the same figure we show the performance for the case of biased 
messages {pf^ = fs = 0.1), at code rate R = 1/4. Also here the agreement with 



Monte-carlo integrations is rather convincing. The third curve in Fig.|10] shows the 
performance for biased messages at Nishimori's temperature T/v, as expected, it is far 
superior compared to low temperature performance and the agreement with Monte-carlo 
results is even better. 



In Fig.|Tl| we show the results obtained for K = 5 and C = 10. For unbiased 
messages the system is extremely sensitive to the choice of initial conditions and does not 
perform well in average even at Nishimori's temperature. For biased messages {fs = 0.1, 
R = 1/4) results are far better and in agreement with Monte-carlo integration of the 
RS saddle-point equations. 
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The experiments show that behef propagation methods may be used successfully for 
decoding Sourlas-type codes in practice, and provide solutions that are well described 
by RS analytical solutions. 

4-3. Basin of Attraction 




Figure 12. Top: Maximum initial deviation A for decoding. Top: A as function of the 
number of interactions K. Circles are averages over 10 different codes with N = 300, 
R— 1/3 and noise level p = 0.1. Bottom: A as function of the connectivity C. Circles 
are averages over 10 codes with N = 300, K = 3 and noise level p = 0.1. Lines and 
X 's correspond to the RS dynamics described by the saddle-point equations. 

To asses the size of the basin of attraction we consider the decoding process as a 
dynamics in the graphs space where edges are considered as dynamical variables. 
In gauged transformed equations ( |53D , the perfect decoding of a message correspond to 
6q^j = 1 . To analyse the basin of attraction we start with random initial values with 
a given normalized deviation from the perfect decoding A = j^J^/iji^ — 
analogous to the finite magnetizations used in the naive mean-field of Section II, since 
a given - corresponds to a given magnetization value by using Eg. (|^) . 



In Fig.|T2| we show the maximal deviation in initial conditions required for successful 
decoding. Top figure shows an average over 10 different codes with = 300 (circles) 
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for a fixed code rate R = 1/3, fixed noise level p = 0.1 and increasing K. Bottom 
figure shows the maximal deviation in initial conditions for a fixed number of spins per 
interaction K = 3, noise level p = 0.1 and increasing C. We confirm the fidelity of the RS 
description by comparing the experimental results with the basin of attraction predicted 
by saddle-point equations (|T^. One can interpret these equations as a dynamics in the 
space of distributions 7r(x). Performing the transformation X = tanh(/3x), one can move 
to the space of distributions n(X) with support over [— 1,+1]. The initial conditions 
can then be described simply as n°(X) = (1 - f )5(X - 1) + + 1). In Fig.|g we 

show the basin of attraction of this dynamics as lines and x 's. 

The K = 2 case is the only practical code from a dynamical point of view, since it 
has the largest basin of attraction and no prior knowledge on the message is necessary 
for decoding. Nevertheless, this code's performance degrades faster than the K > 2 
case as shown in Section III, which points to a compromise between good dynamical 
properties in one side and good performance in the other. One idea could be having a 
code with changing K, starting with A' = 2 to guarantee convergence and progressively 



increasing its values to improve the performance |25 



On the other hand, the basin of attraction increases with C. Again it points to a 
trade off between good equilibrium properties (small C and large code rates) and good 
dynamical properties (large C, large basin of attraction). Mixing small and large C 



values in the same code seems to be a way to take advantage of this trade-off |26, 27, 28 



5. Concluding Remarks 

In this paper we studied, using the replica approach, a finite connectivity many-body 
spin glass that corresponds to Sourlas' codes for finite code rates. We have shown, 
using a simplified one step RSB solution for spin glass phase, that for ^ oo and 
C = aK regime at low temperatures the system exhibits a FERRO-SG phase transition 
that corresponds to Shannon's bound. However, we have also shown that the decoding 
problem for large K has bad convergence properties when simulated annealing strategies 
are used. 

We were able to find replica symmetric solutions for finite K and found good 
agreement with practical decoding performance using belief networks. Moreover, we 
have shown that RS saddle-point equations actually describe the mean behavior of 
belief propagation algorithms. 

We studied the dynamical properties of belief propagation and compared to 
statistical physics predictions, confirming the validity of the description. The basin of 
attraction was shown to depend on K and C. Strategies for improving the performance 
were discussed. 



The same methodology has been recently employed successfully pS] to state-of-the 
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art algorithms as the recent rediscovered Gallager codes |^ and its variations [p5| , [25 
We beheve that the connections found between behef networks and statistical physics 
can be further developed to provide deeper insights into the typical performance of 
general error-correcting codes. 
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Appendix A. Free Energy 



In order to compute free-energies one needs to calculate the replicated partition function 
(p^). One can start from Eq. (|Tl|): 



Tr 



{37} 



exp }) 



(Al) 



where Ti^'^'' {{8°"}) represents the replicated Hamiltonian and a the replica indices. First 
one averages over the parity check tensors A, for that an appropriate distribution has 
to be introduced, denoting /x = (ii, ...^ik) for a specific set of indices: 



{A} t V/^V / 



(A2) 



where the 5 distribution imposes a restriction on the connectivity per spin, A/" is a 
normalization coefficient and the notation means the set /i minus the element i. 
Using integral representations for the delta functions and rearranging: 



(Z") = Tr 



1 



Mm 



dzi 1 



27ri Zi 



c+i 



] E ( 11(11 exp {-m^''\{s-}) 



(A3) 



Remembering that A G {0, 1}, and using the expression (|l]) for the Hamiltonian one 
can change the order of the summation and the product above and sum over A: 



(Z") = Tr 



n 



i+(n^.)exp /5j,5:n^r 



Using the identity exp{(3J^Uie^ S^) = cosh(/5) [l + (n^e;, ^f) tanh(/5 J^)' 
perform the product over a to write: 



(A4) 



one can 
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(A5) 



X 



n 



1 + n cosh"(/?) 1 + {tanh{(3j))jj2 n sr 



{aia2) ie/i je/i y 

Defining (/Xi,/i2, ■■■,IJ'i) as an ordered set of sets, and observing that for large N, 

X](^i...rt>(---) — il "^^^ perform the product over the sets // and replace the 

series that appears by an exponential: 



xexp 



cosh"(/3) I EdI + (tanh(/3 J)), E E n ^^-5,^ 



+ (tanh2(/3j)), E En^^'^r'^r 

Observing that = ^/ defining % = (cosh"(/9J)tanh'(/?J)) j and 
introducing auxiliary variables gai...a„ = jf J2i ^iS^^ ■■■Si"' one finds: 



/^)(n/ 



xexp 



xexp 



dqadqa 
2m 



K\ 



(0102) 



Q:ia2 Q.OC\OL2 

+ ... 



XlV{5«} 

The normalization constant is given by: 



(A7) 



{A} i \n\i J 

and can be computed using exactly the same methods as above resulting in: 



dzj 1 



2m z, 



c+i 



:/ 



2m 



exp 



N- 



K 

-Qq - Nqoqo + go E - 



(A8) 



(A9) 
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Computing the integrals over Zj's and using Laplace's method to compute the 
integrals over go and go one gets: 

(AlO) 

The extremum point is given by go = N^^^^^^^[(K — 1)\CY^^ and go = 
(C N)''^~^^^'^ [(K — . Replacing the Lagrange multipliers in Eg. (|A7| ) using 

gai...a,„/go <lai...a,n a^d qai...a,r, / ^ Qai...a,„, computiug the integrals over Zi and 
using Laplace's method to evaluate the integrals over the Lagrange multipliers one 
finally find Eq.(0). 



J\f = exp < Extr 



90,90 



— go^-Argogo + Arin(|^ 



Appendix B. Replica Symmetric Solution 



The replica symmetric free-energy ([T8|) can be obtained by plugging the ansatz (|T7|) into 
Eg. ([A^) ■ After computing the normalization M and using Laplace's method one has: 



{Z^)A,i,J = exp {iVExtr^- [^^^i - C + 
where: 

K 

01 = % + TiY^ Y[{dxj TT{xj) tanh(/3xj)) 

a j 

+T2 ^ / n ( '^•^i ^(^i) tanh^(/3xj)^ + . . . 



(Bll) 



(B12) 



(aia2> j 



J dxdy7i{x)n{y)tanh^{Px)tanh^{Py) + . . . 



(0102) ' 



and 



a34in{(n/ 



dzj 1 



2m z. 



exp HFY.iS'i 



xexp go XI + ^i'S'f / c/?/7f(?/)tanh(/?y) 



(0l02> « 



(B13) 



(B14) 
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The equation for Qi can be worked out by using the definition of %n and the fact 



that (E(ai...aO 1) = ( I 

Qi = ^cosh"(/3J) J dxjn{xj)j + tanh(/5 J) ntanh(/?, 

Following exactly the same steps one obtains: 
Q2 = j dx dyir^x) vf (y) (1 + tanh(/3x) tanh(/?7/))" , 
and 



(B15) 



(B16) 



In < Tr 



exp (^mT.s'')) 

exp (^qozj dyn{y)f[{l + S'^tanh{Py))^ 



dz 1 



2m z^+i 

Computing the integral over Zi and the trace one finally finds: 



9o 



^3 = In ^ ^ y n dyi'niyi) 



c 



c 



E (e'^^^^) n(l + atanh(/5i/0) 

<T=±l ^ 1 = 1 



(B17) 



(B18) 



Putting everything together, using Eq.(|TOD and some simple manipulation one finds 
Eq.®. 

Appendix C. Zero Temperature Self-consistent Equations 

In this appendix we describe how one can write a set of self-consistent equations to solve 
the zero temperature saddle-point equations (^). Supposing a three peaks ansatz given 
by: 



C-l 



Ax) = ^[P±,Po;C-i](0 ^x - 0, 

l=l-C 



with 

^[p+,po,p-;C] (0 



E 



1^ ^0 P-- 

kl hi ml 



(C19) 
(C20) 

(C21) 



{k,h,Tn ; k—h=l ; A;+/i+m=C— 1} 

One can consider the problem as a random walk, where Ti{y) describes the probability 
of one step of length y [y > means one step to the right) and 7r(x) describes the 
probability of being at distance x from the origin after C — 1 steps. With this idea in 
mind it is relatively easy to understand Tij,^^p^^p_.c-i]{^) as the probability of walking the 
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distance / after C — 1 steps with the probabihties p^, p_ and po of respectively moving 
right, left and staying at the same position. We define the probabilities of walking 
right/left as 'ijj± = J2f~^ r[p+,po,p-;C-i](='=0- Using the second saddle-point equation 
one can write: 



P+ 



K-l 



Yl dxiTx{xi) 



1 1=1 



K-l 



sign(J Jl xz)min(| J |, | Xi |, • • • , | xk-i 

1=1 J / J 



(C22) 



The left side of the above equality can be read as the probability of making K — 1 
independent walks such that after C — 1 steps all of them are not in the origin and an 
even (for J = +1) or odd (for J = —1) number of walks are at the left side. Using this 
reasoning for p_ and po one can finally write : 



p+ = (1 - P) {\]') ^^^-^^-^-^ + P ' ( ; ; ) ^^^-"vr^-^ 

+p^^-^odd{K -1) (C23) 

p- = (1 - P) ( J" ; ; ) ^^^"v?-^-^ + P (\-') 

j=0 ^ ■' ^ j=0 \ ■' / 

+ (l-p)^5-Wd(ir-l), (C24) 

where odd(x) = 1(0) if x is odd (even). Using that p+ + p- + po = 1 one can obtain pq. 
A similar set of equations can be obtained for a five peaks ansatz leading to the same 
set of solutions for the FERRO and PARA phases. The PARA solution po = 1 is always 
a solution, for C > K a FERRO solution with > p_ > emerges. 

Appendix D. 

In this appendix we establish the identity (J)j = (J tanh(/9Ar J)) j for symmetric 
channels. It was shown in ||^ that : 

where is the Nishimori's temperature and p{J \ J^) are the probabilities that a 
transmitted bit J° is received as J. From this we can easily find: 



In a symmetric channel {p{J \ — J°) = p{—J \ J^)), it is also represented as 

tanh {Pj, J) = (D27) 
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Therefore, 



(Jtanh (/5jvJ))j = Trjp(J| 1) 



p{J\l)+p{-J\l) 




+Tij p{J I 1) 



(-J) p{-J\l) 



Trjp(J| 1) 



p{J\l)+p{-J\l) 
Jp{J\l) 



p{J\l)+p{-J\l) 



+Trjp{-J\l) 



Jp{J\l) 



p{-J\l)+p{J\l) 



TrjJp{J\l) = {J)j. 



(D28) 
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